Content in this notebook is modified from content originally written by:
Inspiration from the blog posts “The Illustrated Transformer” and “The Illustrated GPT2” by Jay Alammar, highly recommended reading.
Although the name “language models” is derived from Natural Language Processing, the models used in these approaches can be applied to diverse scientific applications as illustrated below.
During this session I will cover:
Sequences are variable-length lists with data in subsequent iterations that depends on previous iterations (or tokens).
Mathematically:
A sequence is a list of tokens:
T = [t_1, t_2, t_3,...,t_N]
where each token within the list depends on the others with a particular probability:
P(t_N | t_{N-1}, ..., t_3, t_2, t_1)
The purpose of sequential modeling is to learn these probabilities for possible tokens in a distribution to perform various tasks including:
Nucleic acid sequences can be used to predict translation of proteins, mutations, and gene expression levels.
Here is an image of GenSLM. This is a language model developed by Argonne researchers that can model genomic information in a single model. It was shown to model the evolution of SARS-COV2 without expensive experiments.
Protein sequences can be used to predict folding structure, protein-protein interactions, chemical/binding properties, protein function and many more properties.
We will now briefly talk about the progression of language models.
The most common LMs base their design on the Transformer architecture that was introduced in 2017 in the “Attention is all you need” paper.
Since then a multitude of LLM architectures have been designed.
Let’s look at an example of running inference with a LLM as a block box to generate text given a prompt and we will also initiate a training loop for an LLM
Here, we will use the transformers library which is as part of HuggingFace, a repository of different models, tokenizers and information on how to apply these models
Warning: Large Language Models are only as good as their training data.
They have no ethics, judgement, or editing ability.
We will be using some pretrained models from Hugging Face which used wide samples of internet hosted text.
The datasets have not been strictly filtered to restrict all malign content so the generated text may be surprisingly dark or questionable.
They do not reflect our core values and are only used for demonstration purposes.
'''
Uncomment below section if running on sophia jupyter notebook
'''
# import os
# os.environ["HTTP_PROXY"]="proxy.alcf.anl.gov:3128"
# os.environ["HTTPS_PROXY"]="proxy.alcf.anl.gov:3128"
# os.environ["http_proxy"]="proxy.alcf.anl.gov:3128"
# os.environ["https_proxy"]="proxy.alcf.anl.gov:3128"
# os.environ["ftp_proxy"]="proxy.alcf.anl.gov:3128"'\nUncomment below section if running on sophia jupyter notebook\n'[2025-07-24 11:46:18,704651][I][ezpz/__init__:265:ezpz] Setting logging level to 'INFO' on 'RANK == 0' [2025-07-24 11:46:18,706361][I][ezpz/__init__:266:ezpz] Setting logging level to 'CRITICAL' on all others 'RANK != 0'
from transformers import AutoTokenizer,AutoModelForCausalLM, AutoConfig
input_text = "My dog really wanted to eat icecream because"
from transformers import pipeline
generator = pipeline("text-generation", model="gpt2")
pipe = pipeline("text-generation", model="gpt2")
generator(input_text, max_length=20, num_return_sequences=5)[{'generated_text': 'My dog really wanted to eat icecream because it was delicious!"\n\nBut the man was the only one who got to eat his meal. He had been waiting for him for hours, and was already so excited, that he was almost ready to go on his way.\n\nHe knew that the person who had finally eaten his meal was a normal young man, but he didn\'t want to get scared.\n\nHe said to the guy, "I am going to buy you a sandwich. You don\'t want to come back to the village, but I will be in charge of the food for you."\n\nOf course, he didn\'t care.\n\nHe couldn\'t help but feel the anger on his face.\n\nHe was just a normal guy and he didn\'t want to see such a big deal.\n\nThis man wanted to help, and he didn\'t want to become a monster and kill everyone.\n\nAnd for that reason, he didn\'t want to let this man go.\n\n"I\'ll buy you a sandwich. You don\'t want to come back to the village. You can\'t go back to the village if you don\'t want to eat icecream."\n\n"Thank you, I\'ll buy you a sandwich. You don\'t want to come back'},
{'generated_text': 'My dog really wanted to eat icecream because she\'s so good," he told me. "I just thought, \'Wow, this is really, really good.\' So it was a really fun time."\n\nHe said he and his dog had been playing with ice cream and he had put on an ice cream cone to get the kids to enjoy ice cream, and he and his wife had a party to play with ice cream.\n\nWhen I asked what his favorite ice cream was, he said, "The ice cream is awesome."\n\n"People think it\'s all about the ice cream," he said. "It\'s all about the food. I\'m not just a dessert person but I\'m a dessert person and I love ice cream. I love the flavors of it too."\n\nHe said he is a fan of making ice cream out of whole milk, and he loves trying different flavors of the fruit.\n\n"I like the ice cream so much that I made my own," he said. "I love the flavor of the milk, and the flavor of the cream. I don\'t know if it\'s the cream or the milk, but I love the flavor of the ice cream."\n\nHe said he also likes the flavors of fruits, especially strawberries.\n\nHe said he'},
{'generated_text': 'My dog really wanted to eat icecream because I think he wanted to.\n\n"He knew that I didn\'t like ice cream at all. I didn\'t like the taste, so he had to make it himself.\n\n"I think she was scared of the ice cream.\n\n"I don\'t think he\'d do anything to make it worse. He was going to put it up next to my car because I was scared.\n\n"I didn\'t want to do it.\n\n"I was a little bit nervous.\n\n"But I was scared of the ice cream.\n\n"He was like a little boy and he wanted to eat it too.\n\n"He started eating it.\n\n"He didn\'t care about the ice cream and he was eating it."\n\nThe dogs were rescued from the sea after several days in the rough water by the Scottish Sea Rescue.\n\nThey were rescued on January 18 by the Yorkshire Dales with the help of the Scottish Sea Rescue and the Scottish Sea Rescue Dog Rescue.\n\nThe dogs were last seen on April 1.\n\nThe dogs were rescued from the bottom of a canal in Sillam, Devon on March 14.\n\nThey were last seen on January 17 and are being kept by the Royal'},
{'generated_text': 'My dog really wanted to eat icecream because it was something I was trying to avoid. He was doing a couple of flips and we were really close."\n\nThe dog\'s owner, who asked not to be named, said the couple got into an argument in 2009 when they were driving. She went to the gas station and was told the owner had "a dog."\n\n"He was upset and upset for a long time," she said. "It was all for the better."\n\nThe owner said the dog had the same personality as her, and she didn\'t know who it was.\n\nNeighbors said the guy was a nice guy, and that they were looking forward to seeing him again.\n\n"His demeanor was as normal, like he really enjoyed it," Neighbors General Manager, David Sperry said. "He was a nice guy and everyone was very friendly."\n\nNeighbors said the dog had been in their yard for a while, and they\'ve been told they\'re now using his door as a lock.\n\n"They had a little bit of a fight with him, but it was just the best thing we could have done," Sperry said.\n\nNeighbors said the dogs are well-behaved and play a big part in their community'},
{'generated_text': 'My dog really wanted to eat icecream because it was his favorite! He tried to buy a box of chocolate ice cream, but it was too expensive! He tried to buy a box of watermelon ice cream, but it was too expensive! He tried to buy a box of ice cream with a chocolate or jelly flavor. He tried to buy some ice cream with a chocolate flavor! He tried to buy a box of ice cream with a jelly flavor! He tried to buy a box of ice cream with a jelly flavor. He tried to buy a box of ice cream with a jelly flavor. He tried to buy a box of ice cream with a jelly flavor. He tried to buy a box of ice cream with a jelly flavor. He tried to buy a box of ice cream with a jelly flavor. He tried to buy a box of ice cream with a jelly flavor. He tried to buy a box of ice cream with a jelly flavor. He tried to buy a box of ice cream with a jelly flavor. He tried to buy a box of ice cream with a jelly flavor.\n\nHe tried to buy ice cream with a jelly flavor. He tried to buy ice cream with a jelly flavor. He tried to buy ice cream with a jelly flavor. He tried to buy ice cream with a jelly flavor. He tried'}]There are two components that are “black-boxes” here:
Humans can inherently understand language data because they previously learned phonetic sounds.
Machines don’t have phonetic knowledge so they need to be told how to break text into standard units to process it.
They use a system called “tokenization”, where sequences of text are broken into smaller parts, or “tokens”, and then fed as input.
Tokenization is a data preprocessing step which transforms the raw text data into a format suitable for machine learning models. Tokenizers break down raw text into smaller units called tokens. These tokens are what is fed into the language models. Based on the type and configuration of the tokenizer, these tokens can be words, subwords, or characters.
Types of tokenizers:
Let’s look at an example of tokenization using byte-pair encoding.
from transformers import AutoTokenizer
# A utility function to tokenize a sequence and console.log out some information about it.
def tokenization_summary(tokenizer, sequence):
# get the vocabulary
vocab = tokenizer.vocab
# Number of entries to console.log
n = 10
# console.log subset of the vocabulary
console.log("Subset of tokenizer.vocab:")
for i, (token, index) in enumerate(tokenizer.vocab.items()):
console.log(f"{token}: {index}")
if i >= n - 1:
break
console.log("Vocab size of the tokenizer = ", len(vocab))
console.log("------------------------------------------")
# .tokenize chunks the existing sequence into different tokens based on the rules and vocab of the tokenizer.
tokens = tokenizer.tokenize(sequence)
console.log("Tokens : ", tokens)
console.log("------------------------------------------")
# .convert_tokens_to_ids or .encode or .tokenize converts the tokens to their corresponding numerical representation.
# .convert_tokens_to_ids has a 1-1 mapping between tokens and numerical representation
# ids = tokenizer.convert_tokens_to_ids(tokens)
# console.log("encoded Ids: ", ids)
# .encode also adds additional information like Start of sequence tokens and End of sequene
console.log("tokenized sequence : ", tokenizer.encode(sequence))
# .tokenizer has additional information about attention_mask.
# encode = tokenizer(sequence)
# console.log("Encode sequence : ", encode)
# console.log("------------------------------------------")
# .decode decodes the ids to raw text
ids = tokenizer.convert_tokens_to_ids(tokens)
decode = tokenizer.decode(ids)
console.log("Decode sequence : ", decode)
tokenizer_1 = AutoTokenizer.from_pretrained("gpt2") # GPT-2 uses "Byte-Pair Encoding (BPE)"
sequence = "Counselor, please adjust your Zoom filter to appear as a human, rather than as a cat"
tokenization_summary(tokenizer_1, sequence)[11:46:46] Subset of tokenizer.vocab: ĠLots: 25024 TP: 7250 Ġcaptain: 10654 pipe: 34360 loop: 26268 Ġ302: 32591 Ġcrumble: 49478 Ġupwards: 21032 screen: 9612 ĠFred: 8559 Vocab size of the tokenizer = 50257 ------------------------------------------ Tokens : ['Coun', 'sel', 'or', ',', 'Ġplease', 'Ġadjust', 'Ġyour', 'ĠZoom', 'Ġfilter', 'Ġto', 'Ġappear', 'Ġas', 'Ġa', 'Ġhuman', ',', 'Ġrather', 'Ġthan', 'Ġas', 'Ġa', 'Ġcat'] ------------------------------------------ tokenized sequence : [31053, 741, 273, 11, 3387, 4532, 534, 40305, 8106, 284, 1656, 355, 257, 1692, 11, 2138, 621, 355, 257, 3797] Decode sequence : Counselor, please adjust your Zoom filter to appear as a human, rather than as a cat
Words are turned into vectors based on their location within a vocabulary.
The strategy of choice for learning language structure from tokenized text is to find a clever way to map each token into a moderate-dimension vector space, adjusting the mapping so that
Similar, or associated tokens take up residence nearby each other, and different regions of the space correspond to different position in the sequence. Such a mapping from token ID to a point in a vector space is called a token embedding. The dimension of the vector space is often high (e.g. 1024-dimensional), but much smaller than the vocabulary size (30,000–500,000).
Various approaches have been attempted for generating such embeddings, including static algorithms that operate on a corpus of tokenized data as preprocessors for NLP tasks. Transformers, however, adjust their embeddings during training.
Now let’s look at the base elements that make up a Transformer by dissecting the popular GPT2 model
GPT2LMHeadModel( (transformer): GPT2Model( (wte): Embedding(50257, 768) (wpe): Embedding(1024, 768) (drop): Dropout(p=0.1, inplace=False) (h): ModuleList( (0-11): 12 x GPT2Block( (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (attn): GPT2Attention( (c_attn): Conv1D(nf=2304, nx=768) (c_proj): Conv1D(nf=768, nx=768) (attn_dropout): Dropout(p=0.1, inplace=False) (resid_dropout): Dropout(p=0.1, inplace=False) ) (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (mlp): GPT2MLP( (c_fc): Conv1D(nf=3072, nx=768) (c_proj): Conv1D(nf=768, nx=3072) (act): NewGELUActivation() (dropout): Dropout(p=0.1, inplace=False) ) ) ) (ln_f): LayerNorm((768,), eps=1e-05, elementwise_affine=True) ) (lm_head): Linear(in_features=768, out_features=50257, bias=False) )
GPT2 is an example of a Transformer Decoder which is used to generate novel text.
Decoder models use only the decoder of a Transformer model. At each stage, for a given word the attention layers can only access the words positioned before it in the sentence. These models are often called auto-regressive models. The pretraining of decoder models usually revolves around predicting the next word in the sentence.
These models are best suited for tasks involving text generation.
The architecture of GPT-2 is inspired by the paper: “Generating Wikipedia by Summarizing Long Sequences” which is another arrangement of the transformer block that can do language modeling. This model threw away the encoder and thus is known as the “Transformer-Decoder”.
Key components of the transformer architecture include:
Input Embeddings: Word embedding or word vectors help us represent words or text as a numeric vector where words with similar meanings have the similar representation.
Positional Encoding: Injects information about the position of words in a sequence, helping the model understand word order.
Self-Attention Mechanism: Allows the model to weigh the importance of different words in a sentence, enabling it to effectively capture contextual information.
Feedforward Neural Networks: Process information from self-attention layers to generate output for each word/token.
Layer Normalization and Residual Connections: Aid in stabilizing training and mitigating the vanishing gradient problem.
Transformer Blocks: Comprised of multiple layers of self-attention and feedforward neural networks, stacked together to form the model.
Since attention mechanisms are arguably the most powerful component of the Transformer, let’s discuss this in a little more detail.
Suppose the following sentence is an input sentence we want to translate using an LLM:
”The animal didn't cross the street because it was too tired”
To understand a full sentence, the model needs to understand what each word means in relation to other words.
For example, when we read the sentence: ”The animal didn't cross the street because it was too tired” we know intuitively that the word "it" refers to "animal", the state for "it" is "tired", and the associated action is "didn't cross".
However, the model needs a way to learn all of this information in a simple yet generalizable way. What makes Transformers particularly powerful compared to earlier sequential architectures is how it encodes context with the self-attention mechanism.
As the model processes each word in the input sequence, attention looks at other positions in the input sequence for clues to a better understanding for this word.
In practice, multiple attention heads are used simultaneously.
This: * Expands the model’s ability to focus on different positions. * Prevents the attention to be dominated by the word itself.
We are going to use the powerful visualization tool bertviz, which allows an interactive experience of the attention mechanisms. Normally these mechanisms are abstracted away but this will allow us to inspect our model in more detail.
Let’s load in the model, GPT2 and look at the attention mechanisms.
Hint… click on the different blocks in the visualization to see the attention
from transformers import AutoTokenizer, AutoModel, utils, AutoModelForCausalLM
from bertviz import model_view
utils.logging.set_verbosity_error() # Suppress standard warnings
model_name = 'openai-community/gpt2'
input_text = "No, I am your father"
model = AutoModelForCausalLM.from_pretrained(model_name, output_attentions=True)
tokenizer = AutoTokenizer.from_pretrained(model_name)
inputs = tokenizer.encode(input_text, return_tensors='pt') # Tokenize input text
outputs = model(inputs) # Run model
attention = outputs[-1] # Retrieve attention from model outputs
tokens = tokenizer.convert_ids_to_tokens(inputs[0]) # Convert input ids to token strings
model_view(attention, tokens) # Display model viewNow, let’s see a practical application of LLMs using a HuggingFace pipeline for classification.
This involves a few steps including: 1. Setting up a prompt 2. Loading in a pretrained model 3. Loading in the tokenizer and tokenizing input text 4. Performing model inference 5. Interpreting inference output
A “prompt” refers to a specific input or query provided to a language model. They guide the text processing and generation by providing the context for the model to generate coherent and relevant text based on the given input.
The choice and structure of the prompt depends on the specific task, the context and desired output. Prompts can be “discrete” or “instructive” where they are explicit instructions or questions directed to the language model. They can also be more nuanced by more providing suggestions, directions and contexts to the model.
We will use very simple prompts in this tutorial section, but we will learn more about prompt engineering and how it helps in optimizing the performance of the model for a given use case in the following tutorials.
The AutoModelForSequenceClassification from_pretrained() method instantiates a sequence classification model.
Refer to https://huggingface.co/transformers/v3.0.2/model_doc/auto.html#automodels for the list of model classes supported.
“from_pretrained” method downloads the pre-trained weights from the Hugging Face Model Hub or the specified URL if the model is not already cached locally. It then loads the weights into the instantiated model, initializing the model parameters with the pre-trained values.
The model cache contains:
[11:46:50] DistilBertConfig { "activation": "gelu", "architectures": [ "DistilBertForSequenceClassification" ], "attention_dropout": 0.1, "dim": 768, "dropout": 0.1, "finetuning_task": "sst-2", "hidden_dim": 3072, "id2label": { "0": "NEGATIVE", "1": "POSITIVE" }, "initializer_range": 0.02, "label2id": { "NEGATIVE": 0, "POSITIVE": 1 }, "max_position_embeddings": 512, "model_type": "distilbert", "n_heads": 12, "n_layers": 6, "output_past": true, "pad_token_id": 0, "qa_dropout": 0.1, "seq_classif_dropout": 0.2, "sinusoidal_pos_embds": false, "tie_weights_": true, "transformers_version": "4.53.3", "vocab_size": 30522 }
Here, we load in a pretrained tokenizer associated with this model.
[11:46:51] tensor([[ 101, 1996, 6090, 6525, 7712, 3193, 1997, 1996, 4153, 2001, 3052, 17904, 1012, 102]])
Here, we: * load data into the model, * perform inference to obtain logits, * Convert logits into probabilities * According to probabilities assign label
The end result is that we can predict whether the input phrase is positive or negative.
# STEP 5 : Perform inference
outputs = model(input_ids)
result = outputs.logits
console.log(result)
# STEP 6 : Interpret the output.
probabilities = F.softmax(result, dim=-1)
console.log(probabilities)
predicted_class = torch.argmax(probabilities, dim=-1).item()
labels = ["NEGATIVE", "POSITIVE"]
out_string = "[{'label': '" + str(labels[predicted_class]) + "', 'score': " + str(probabilities[0][predicted_class].tolist()) + "}]"
console.log(out_string)tensor([[-4.2767, 4.5486]], grad_fn=<AddmmBackward0>) tensor([[1.4695e-04, 9.9985e-01]], grad_fn=<SoftmaxBackward0>) [{'label': 'POSITIVE', 'score': 0.9998530149459839}]
Model can be saved and loaded to and from a local model directory.
from transformers import AutoModel, AutoModelForCausalLM
# Instantiate and train or fine-tune a model
model = AutoModelForCausalLM.from_pretrained("bert-base-uncased")
# Train or fine-tune the model...
# Save the model to a local directory
directory = "my_local_model"
model.save_pretrained(directory)
# Load a pre-trained model from a local directory
loaded_model = AutoModel.from_pretrained(directory)[2025-07-24 11:46:52,246] [INFO] [real_accelerator.py:254:get_accelerator] Setting ds_accelerator to mps (auto detect)huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
- Avoid using `tokenizers` before the fork if possible
- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
W0724 11:46:52.520000 41975 torch/distributed/elastic/multiprocessing/redirects.py:29] NOTE: Redirects are currently not supported in Windows or MacOs.[2025-07-24 11:46:53,327] [INFO] [logging.py:107:log_dist] [Rank -1] [TorchCheckpointEngine] Initialized with serialization = FalseThe Model Hub is where the members of the Hugging Face community can host all of their model checkpoints for simple storage, discovery, and sharing.
Model Hub Link : https://huggingface.co/docs/hub/en/models-the-hub
Example of a model card : https://huggingface.co/bert-base-uncased/tree/main
google/gemma-2-2b-itmicrosoft/Phi-3-mini-4k-instructmeta-llama/Llama-3.2-1Bgpt2 if having trouble loading these models in@online{foreman2025,
author = {Foreman, Sam},
title = {Hands {On:} {Introduction} to {Large} {Language} {Models}
{(LLMs)}},
date = {2025-07-23},
url = {https://saforem2.github.io/hpc-bootcamp-2025/02-llms/1-hands-on-llms/},
langid = {en}
}